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Abstract 

Background: Despite growing interest and urges by leading experts for tlie routine collection of patient reported 
outcome (PRO) measures in all general care patients, and in particular cancer patients, there has not been an 
updated comprehensive review of the evidence regarding the impact of adopting such a strategy on patients, 
service providers and organisations in an oncologic setting. 

Methods: Based on a critical analysis of the three most recent systematic reviews, the current systematic review 
developed a six-method strategy in searching and reviewing the most relevant quantitative studies between 
January 2000 and October 201 1 using a set of pre-determined inclusion criteria and theory-based outcome 
indicators. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) system was used 
to rate the quality and importance of the identified publications, and the synthesis of the evidence was conducted. 

Results: The 27 identified studies showed strong evidence that the well-implemented PROs improved patient- 
provider communication and patient satisfaction. There was also growing evidence that it improved the monitoring 
of treatment response and the detection of unrecognised problems. However, there was a weak or non-existent 
evidence-base regarding the impact on changes to patient management and improved health outcomes, changes 
to patient health behaviour, the effectiveness of quality improvement of organisations, and on transparency, 
accountability, public reporting activities, and performance of the health care system. 

Conclusions: Despite the existence of significant gaps in the evidence-base, there is growing evidence in support 
of routine PRO collection in enabling better and patient-centred care in cancer settings. 
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Background 

Patient reported outcome (PRO) measures include 
health status assessments and measures for health- 
related quality-of-life (HRQOL), symptom reporting, 
satisfaction with care, treatment satisfaction, economic 
impact, and specific dimensions of patient experience 
such as depression and anxiety [1]. The USA Food and 
Drug Agency (FDA) adopts a much broader definition 
[2] as "A PRO is any report coming directly from patients 
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about a health condition and its treatment", meaning 
that PROs capture patients' perspectives about how ill- 
ness or new therapies impact on their general well-being 
. There is a growing interest from clinicians, researchers, 
industry and policy-makers in routinely collecting PROs 
to facilitate timely, patient-centred and evidence-based 
care. For example, the National Health Service (NHS) of 
the UK has been implementing a world-leading initiative 
for the routine collection of PROs that firstly included 
a few selected elective surgeries (e.g. unilateral 
hip replacements, unilateral knee replacements, groin 
hernia surgery or varicose vein surgery) [3] but are soon 
expanding to many other conditions such as mastectomy 
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and breast cancer, among others. In the USA, the 
Patient-Reported Outcomes Measurement Information 
System (PROMIS), a National Institutes of Health 
funded initiative starting in 2004, is providing a publicly 
available web-based resource that can be used to meas- 
ure key health symptoms and HRQOL [4]. The trad- 
itional paper-based PROs instruments are limited by its 
lack of flexibility, language and literacy requirement, 
[5,6] possible inappropriateness towards minority 
groups, [7,8] lack of timeliness (in generating instantan- 
eous clinical meaningful interpretations) [9] and inability 
to adopt state-of-the-art measurement science such as 
Item Response Theory (IRT) and Computer Adapted 
Test (CAT) technique [10]. To overcome the difficulty 
of integrating the administration and analysis of PRO in- 
struments into clinical practice, researchers are develop- 
ing and validating alternatives to traditional paper- 
based instruments such as office-based touch-screen 
computers, [11-13] telephone-based interactive voice- 
response (IVR) systems, [14-16] hand-held computers, 
[17,18] mobile phones, [19-21] and more recently, the 
Internet [22-24]. Some rationales [25-28] put forward 
for measuring PROs in a cancer setting include, but not 
limited to: 1) better communication and shared decision 
making by patients and providers; 2) assessing the health 
status of patients entering therapy and identifying treat- 
able problems; 3) determining the degree and sources 
of the patient's decreased ability to function; 4) 
distinguishing between types of problems, including 
physical, emotional, and social; 5) detecting adverse ef- 
fects of therapy; 6) monitoring the effects of disease pro- 
gression and response to therapy; 7) informing decisions 
about changing treatment plans, and 8) predicting the 
course of disease and outcomes of care. 

However, despite growing interest and urges by the lead- 
ing experts for applying routinely collected PROs for all 
cancer patients, there has not been an updated compre- 
hensive review of the evidence regarding the impact of 
adopting such a strategy on patients, services providers 
and organisations. The most recent review focused only 
on clinical trial design [26] studies of cancer patients, and 
only assessed a limited number of outcomes. The current 
project aims to provide the much needed comprehensive 
review update, including all relevant quantitative studies 
investigating the effectiveness of routine PRO collection in 
cancer patients. The review research questions were: 

1. What are the impacts of composite measures of 
PROs collected on cancer patients during treatment 
with regards to: 

a) Provider behaviour for improving care delivered; 

b) Organisational changes within health care settings 
for improving processes and models of care 

(e.g. targeting and tailoring care); 



c) Improving clinical outcomes for patients; and 

d) Improving patient experience of care (e.g. self-care). 

2. What mechanisms are involved in the link between 
PROs and the impacts identified in 1(a)? 

3. What factors moderate the extent of the impacts 
identified in 1(a)? 

Methods 

Existing systematic reviews and rationale for the current 
review 

In order to develop an efficient search and review strat- 
egy, over 200 existing reviews on the same or similar 
topics were firstly systematically examined (identified in 
a broad search covering PROs and quality of life mea- 
sures between January 2000 and October 2011). Three 
reviews [26-28] were identified as the baseline reviews 
for this project and their review strategies were carefully 
examined in aspects such as the aim and scope, time 
span, search strategy and search terms used, articles in- 
cluded in each review, and conclusions drawn. A table 
summarising the three systematic reviews is presented in 
Table 1. 

Review search strategy 

Analysing the results of above three systematic reviews 
demonstrates the importance of search strategies in de- 
termining what literature will be included in the study, 
which in turn, may influence what conclusions will be 
derived. Valderas et al.'s (2008) [27] review excluded 
three out of the five clinical trials on cancer patients that 
were included in Marshall et al.'s (2006) [28] review. 
Lucket et al.'s (2009) review [26] excluded one article 
(Taenzer et al. (2000), [33] a before-after study) from 
Marshall et al.'s review [27]. A mixed methodology 
search was developed in order to maximise the identifi- 
cation of recent literature in a short period of time. The 
search was conducted in six different ways as follows: 

1. A text-based search strategy was developed based on 
previous reviews. To elicit previous reviews, a search 
was conducted for the text terms 'patient reported 
outcome*', 'self-reported', 'self-assessed' anywhere in 
title, abstract and key words, combined with 'quality 
of life', 'symptom^ 'functional status', 'health status! 
'patient satisfaction^ 'unmet need*' anywhere in title, 
abstract and key words. For original articles, a 
search was conducted using the same strategy as 
above but restricted to those with 'neoplasm' or 
'cancer' in the key words. The search results were 
restricted to between January 2000 and October 
2011 (full search strategy is listed in Additional file 1: 
Appendix 1). 

2. All reviews were evaluated (over 200 in total on 
various topics but not limited to only cancer 



Table 1 A comparison of three baseline reviews 



First author, 
year 

Luckett et al. 
2009 [26] 



Valderas et al. 
2008 [27] 



Marshall et al. 
2006 [28] 



Aim and review scope 

To identify future strategies for (1) 
interventions to impact patient 
outcomes; and (2) trials to identify 
treatment effects. 



Time span and the search strategy Search Terms 



MEDLINE and PsyclNFO were 
systematically searcfied to identify 
reports of relevant randomised 
controlled trials. The time span was 
between 2006 and 1 August 2008. 
Four cancer trials were cited in a 
previous review (Valderas et al. 2008) 
[27]. 



To summarize the best evidence 
regarding the impact of providing 
patient reported outcomes (PRO) 
information to health care 
professionals in daily clinical practice. 



To synthesize the evidence for using 
publically reported performance data 
to improve quality. Only articles that 
provided empirical evidence on the 
impact of public reporting on 
outcomes (effectiveness, patient 
safety, and patient-centeredness) and 
unintended consequences, as well as 
selection and quality improvement 
activity were included. 



Systematic review of randomised clinical 
trials (Medline, Cochrane Library); 
reference lists of previous systematic 
reviews; and requests to authors and 
experts in the field. Time span: Articles 
published between 1978 and 2007. 



Webspirs Medline was searched for 
the years from January 1976 to 
November 2004. Reference lists of 
included studies and appropriate 
reviews (Greenhaigh & Meadows 1999 
[30]; Espallargues et al. 2000 [29]; 
Gilbody et al. 2003 [31]) were also 
searched for relevant articles. Finally, 
PubMed's 'related articles' feature was 
used with several background and 
included articles (Drur/ et al. 2000 [32]; 
Velikova et al. 2004 [13]) to identify 
publications with a high proportion of 
similar text in the title and abstract. 



1. Examined the citations of the four 
trials 2. Adopted the strategy used by 
Valderas et al. [27] and Espallargues 
et al. [29] which involved searching for 
the terms 'health status', 'functional 
status' or 'quality of life' and 'clinical 
practice', 'clinical setting', 'practice 
setting', 'medical practice' or 'medical 
consultation' anywhere in the title, 
abstract or keywords. Results were 
limited by publication date (2006- 
2008) and the MeSH or keyword 
neoplasm. 

No exact search terms provided but 
indicated available from the author 
upon request. 



Articles included in 
the review 

6 RCTs 



Terms used in relation to patient- 
reported outcome measures (for 
example, 'self report* near2 
measure*') joined with an 'and' 
command to terms related to routine 
practice outcomes (for example, 
'improve* near detect*') or patient 
involvement in the health care 
process (such as 'patient* near 
provider* near interaction*'). 



34 articles 

corresponding to 28 
original studies; only 2 
(not 4) as mentioned in 
the above review, are in 
an oncologic setting. 



40 articles included in 
the review including 5 
publications from an 
oncologic setting. 



Major conclusions 

Future interventions should motivate 
and equip health professionals to use 
PROs data in managing patients, 
training patients in self-efficacy, using 
more specific PROs in clinics, 
improving the interpretability of 
feedback for both medical staff and 
patients, and monitoring the use of 
PROs to intervene when problems 
arise. Future trials should use a cluster 
randomised design to control for 
contamination and enable systems- 
based interventions. 

IVIethodological concerns limit the 
strength of inference regarding the 
impact of providing PROs 
information to clinicians. Results 
suggest great heterogeneity of 
impact; contexts and interventions 
that will yield important benefits 
remain to be clearly defined. 

The pattern of results suggests a 
general lack of clarity in the field, 
especially regarding appropriate goals 
for PROs and the mechanisms by 
which they might achieve them. To 
fully evaluate their role in routine 
practice, studies need to use PROs that 
capture issues of importance to 
patients and to measure impacts 
relating to the patient-provider 
relationship and patient contributions 
to their well-being. Until studies 
evaluate PROs as a means to facilitate 
patient-centred care, their full potential 
in clinical practice will remain unknown. 



Chen ef al. BMC Health Services Research 201 3, 1 3:21 1 
http://www.biomedcentral.com/1472-6963/13/211 



Page 4 of 24 



patients) with three baseline reviews used as the 
starting point for our top-down and bottom-up search 
strategy. We chose the three baseline reviews because 
that: 1) they are all systematic reviews that could be 
helpful in forming the structure or strategy of the 
current review (but not necessarily restricted to cancer 
patients); and 2) they were published after 2005. 

3. All articles were examined if they cited the 7 key 
randomised controlled trials [33-39] listed in the 
above reviews (bottom-up approach). References 
were also sought from the most recently published 
trials, editorials, and commentaries (a top-down 
approach). The powerful citation tracking feature of 
Scopus™ made this strategy feasible. 

4. Simplified text terms (i.e. patient reported outcome, 
PRO, PROM, Quality of life, QOL) were used to 
conduct a web search for identifying grey literature. 

5. Leading researchers and experts in the field (elicited 
through the advice of Cancer Institute NSW (CINSW), 
editorials, review articles and most cited articles) were 
purposefully searched in order to analyse the references 
and citations in their publications. 

6. Some key cancer centres' websites were also 
searched in order to get more detailed information. 

The search was limited to the Scopus™ database as it is 
the largest abstract and citation database of peer-reviewed 
literature and quality web sources including 100% cover- 
age of Medline titles and EBASE. It also tracks, analyses 
and visualises publication results, which is well suited to 
our top-down and bottom-up search strategy. 

Aim, study selection and endpoints of the review 

In this review, the aim was to synthesize the evidence in 
relation to the impact of routinely collected PROs on pa- 
tients, providers, and health organisations. The frame- 
works proposed by Greenhalgh and colleagues [25] and by 
Abernethy and colleagues [40] were adopted to guide our 
evaluation of the existing literature. Greenhalgh et al. [25] 
proposed a framework (Figure 1) that depicts mechanisms 
between the routine collection of PROs and changes in 
patient outcomes. The authors suggest that the multilayer 
mediators (i.e. changes to doctor-patient communication, 
monitoring treatment responses, detecting unrecognised 
problems, changes to patient health behaviour, changes to 
clinicians' management plans, and improved patient satis- 
faction) have complex relationships among them. The 
studies that revealed these complex relationships may as- 
sist in understanding whether and how the underlying 
mechanisms of routinely collected PROs work to improve 
the intended outcomes. 

Recently, Abernethy and colleagues [40] have argued 
that the routine collection of PROs has the capacity to 
impact not only at the patient-level, but by addressing 



the logistics of data linkage, and could ensure that the 
system will grow to accommodate other clinical- and 
health system-level issues; for example, evaluating com- 
parative effectiveness of treatments, monitoring quality 
of care, and translating basic science findings into clin- 
ical practice (Figure 2). The integration of data systems 
will fuel rapid learning cancer care at the national and 
societal levels (see Figure 2a and b), making many types 
of research and system learning possible across institu- 
tions and health sectors. The benefits and implications 
of such a rapid learning health care system may include, 
but is not limited to, strong and effective quality im- 
provement (QI), increased transparency, accountability, 
public reporting, better health system performance 
(monitoring, planning, financing, evaluating, responding) 
and better quality of care. 

Combining both frameworks, a list of outcome indica- 
tors was developed (Table 2) against which each eligible 
study was assessed. To include not only the doctors' ex- 
perience with patients after collecting PROs, but also the 
experience of other health services providers (i.e. nurses, 
allied health workers), the term 'Patient-provider commu- 
nication' was used instead of 'doctor patient communica- 
tion' as proposed by Greenhalgh et al. [25]. In order to 
answer review questions 2 & 3 for the studies included, all 
possible explicit mediation effects were reviewed through 
examining if a path-analysis or a mediation-analysis by 
multiple, staged regression approach was presented in the 
paper. To examine potential moderating effect, each study 
was examined to determine if it explicitly tested the inter- 
action effect/moderating effect, or in explicitly conducted 
subgroup analysis. Significant possible mediating or mod- 
erating effect results were indicated as part of review end- 
points in Table 3. Inferences made and the discussion 
were based on these results. 

Inclusion and exclusion criteria 

The inclusion criteria were: 1) substantial content in pre- 
senting empirical evidence on the impact of routinely col- 
lected PROs on at least one of the outcomes listed in 
Table 2; 2) adult cancer patients; 3) conducted in an onco- 
logic setting including inpatient, outpatient and outreach 
services; and 4) studies using a composite PRO. We defined 
a composite PRO as those PROs are often based on a well- 
developed instrument and with an aim for measuring a 
substantial aspect of patient conditions (or treatment) with 
at least 4 items. To reflect the demanding and complex na- 
ture of evaluating the impact of routine collected PROs, eli- 
gible studies included a variety of designs including, but not 
limited to, randomised controlled trials (RCTs), controlled 
before-after trials (CBA) and interrupted time series (ITS). 
ITS designs have a longitudinal character, with repeated 
measurements and at least three data points before and 
after the intervention point. Surveys and clinical audits 
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PROVISION OF INFORMATION FROM HEALTH STATUS 
MEASURES TO CLINICIANS 

• Alone or supplemented with management guidelines 

• With or without training in interpretation of scores 

• Fed back once or several limes 

• Graphical displays of scores or written summaries, with or without 
DODulation norms, with or without orevious scores 



B 

Evidence: + 

CHANGES TO DOCTOR PATIENT COMMUNje^TION 

• Clinicians talk to patients about their feelings^alth status 

• Develop a shared view of treatment goals/Health status/reason for the visit 



C 

Evidence: +/- 

MONITOR TREATMENT 
RESPONSE 



D 

Evidence: +/- 

DETECT UNRECOGNISED 
PROBLEMS 



E 

Evidence: - 

CHANGES TO PATIENT 
HEALTH BEHAVIOUR 

• Visit to the clinician 

• Adherence to treatment 




F 

Evidence: - 

CHANGES TO CLINICIAN'S 
MANAGEMENT OF PATIENT 

• Changes to/initiation of treatment 

• Referrals to other agencies 

• Tests to investigate problem further 
Advice on problem management 



G 

Evidence: - 

IMPROVED PATIENT SATISFACTION 



A 

Evidence: + 

IMPROVED HEALTH 
OUTCOMES 



Figure 1 A hypothetical framework to understand the impact of routinely collected PROs on patient health outcomes (adopted from 
Greenhaigh et al. (2005) [25] with permission). 



were also included if the studies provided quantitative re- 
sults relevant to the listed outcomes. 

Studies were excluded if they were non-English lan- 
guage articles, opinion and theoretical articles, historical 
descriptions, review articles, feasibility studies of some 
PROs collection devices, studies investigating child 



cancer patients or qualitative studies with no substantial 
quantitative results on the review endpoints. 

Data extraction and quality assessment 

Electronic search results were downloaded into EndNote 
bibliographic software. Two reviewers independently 



wafahouMig or IwMraM mo 
ThaVeY«ieiT)«ni4[xlien-l» 



i 




Figure 2 (a) A data linkage framework (b): A learning health care system. Note: Figures 2: adopted from Aberthnethy et al. (2010) [40] 
with permission. 
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Table 2 Outcome indicators assessed for each eligible 
study included in the review 

Number Outcomes 



1 Patient-provider communication 

2 Monitor treatment response 

3 Detect unrecognised problems 

4 Changes to patient health behaviour 

5 Changes to patient management 

6 Improved patient satisfaction 

7 Improved health outcomes 

8 Strong & effective quality improvement 

9 Increased transparency, accountability, public reporting 

10 Better system performance (monitoring, planning, financing; 
evaluating, responding) 

1 1 Mediating variables on the effect (both at individual and 
organisation level) 

12 Possible subgroup effects 



Note: Both 1 1 & 12 are combined in the summarising tables as few studies 
have explored such issues. 

(JC, LO) screened all titles and abstracts of citations 
identified by the electronic search, applied the selection 
criteria to potentially relevant papers, and extracted data 
from included studies using a standardised form. Any 
disagreements concerning studies to be included were 
resolved by consensus. 

All studies were classified into two domains. Domain 1 
correlated sample characteristics with population wide 
characteristics, and Domain 2 focused on study design. 
The data extraction form was adapted from other review 
studies using the outcome measures discussed above 
(see Table 4). For each eligible study, a list was made in- 
cluding the leading author, country and jurisdiction, de- 
sign, sample, outcome measures, the PROs used, times 
of feedback and intervention, members of medical teams 
given feedback, management plans offered to teams, and 
training (see Table 5). All qualifying studies were listed 
chronologically with the outcome indicators (see 
Table 3). 

In Domain 1, the routinely collected PROs in particu- 
lar participants or samples was classified as rated on a 4- 
point scale representing how closely the participants or 
samples overlapped with the characteristics and needs of 
the intended study populations(l star=very weakly re- 
lated to 4 stars =very strong related). For example, for a 
study conducted in the US on a sample of lung cancer 
patients, the degree of overlap of the study sample with 
the characteristics of lung cancer patients in the US over- 
all was assessed by considering the study setting, sample 
size and sampling frame, response rate, loss-to-follow-up, 
and characteristics of the study sample. In Domain 2, 
study design was classified and rated on 4 categories with 
1 star indicating the weakest design and 4 stars indicating 



the strongest design. Four stars indicated a randomised 
trial or experimental study; 3 stars indicated a controlled 
trial, pre-post trial with control (controlled before-after 
trial), time series, or observational cohort with multivari- 
able adjustment; 2 stars indicated a pre-post trial without 
control, observational cohort study without multivariable 
adjustment, cross-sectional study without multivariable 
adjustment, analysis of time trends without control, or 
well-designed qualitative study; and 1 star indicated a case 
series, other qualitative study, or survey (descriptive) 
study. 

Revised appraisal criteria were adapted from the guide- 
lines on the assessment of quality improvement inter- 
ventions [58,59]. A global rating was also created using 
the Grading of Recommendations, Assessment, Develop- 
ment, and Evaluation (GRADE) system [60]. The British 
Medical Journal has recommended the GRADE system 
since 2006 for grading evidence when submitting a clin- 
ical guidelines article. It has multiple advantages and is 
useful for systematic reviews and health technology as- 
sessments, as well as for evaluating research on clinical 
guidelines. The global rating created in the current study 
was based on the integration of the Domain 1 and Do- 
main 2 ratings, as well as the intervention fidelity (the 
degree of success of the interventional strategy, the pa- 
tients' and providers' adherence to the intervention strat- 
egy), dose-response gradient, precision and validity of 
outcomes (potential confounding factors and biases), 
and uncertainty of the direction of results. The global 
rating was divided into three categories; indication that 
the study should carry great (3 checks), moderate (2 
checks), or little (1 check) weight when considering the 
strength of evidence (see Table 4). Any experimental re- 
search that is reported in the manuscript was performed 
with the approval of an appropriate ethics committee. 

No attempt was made to quantitatively synthesize the re- 
sults as the data were too heterogeneous to support pooling. 

Results 

The multi-method search strategy yielded 27 publica- 
tions that were eligible to be included in the review - a 
significant increase compared with most of the recent 
reviews. The detailed flowchart of the search strategies 
and its relative results was presented (Figure 3). The re- 
sults and conclusions drawn were based on the 27 stud- 
ies included in the analyses despite the large number of 
full-text articles extracted. 

Of the 27 publications, 16 were identified as randomised 
controlled trials, 2 as before-after studies and 9 observa- 
tional studies with 11 studies published before 2009. The 
characteristics and quality of the studies are presented in 
Table 5 with their impact on outcome indicators presented 
in Table 3. As Trowbridge et al. (1997) was the only 
article in the 1990s included in two of the previous 



Table 3 The characteristics of design and study quality 
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Country / 
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Design 
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plan offered 
to team 



Training to 
staff 



Domain 
1 



Domain 
2 



Global 
Rating 



One 



Doctors 
only (12) 



No 



Once 



One 



Doctor and 
coordination 
nurse (numbers 
not reporter) 



No 



Doctors and No 
nurses 



Ground round 
introduction 
and training 



VV 



Individualised 
plan 

developed by 

coordination 

nurse in 

accordance 

with generic 

psychosocial 

guidelines 



No 



VV 



nurse 



Table 3 The characteristics of design and study quality (Continued) 



Detmar et al. 
(2002) [37] 



Mooney 
et al. (2002) 
[16] 



Velikova 
et al. (2004) 
[36] 



Netherland 


RCT: 


214 palliative 


Patient-doctor 


(Netherlands 


(Cross-over 


cancer patients 


communication 


Cancer Institute, 


design) 


In a outpatient 


Doctor's 


Amsterdam) 


Intervention/ 


clinic of a 


awareness of 




control 


cancer hospital 


patient HRQoL 








Patient 








management 
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USA (University 


A pilot 


27 patients 


Telephone-Linked 


of Utah, Salt 


Prospective 


receiving 


Care system for 


Lake City, Utah) 


study over a 


cancer 


Chemotherapy 




month period 


chemotherapy 


(TLC-Chemo 




with daily 


at a cancer 


Alert) Seven 




measures 


centre 


symptoms 
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(r\zi\ icdii :inH 
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UK (Cancer 


RCT: 


286 cancer 


Patient HRQoL 


Research UK 


hterventlon 


outpatients 


(FACT-G) 


Clinical Centre 


/control- 


attending a 


Discussion of 


- Leeds) 


attention/control 


large cancer 


HRQoL Issues In 




in a ratio of 2:1:1 


centre of a 


consultation 






teaching 


IVledlcal actions 






hospital 


(decisions on 








cancer treatment. 








symptomatic/ 








supportive 








treatment. 








Investigations and 








referrals) Non- 








medical actions 








(advice on lifestyle, 








copying and 








reassurance) 








Physician checklist 








assessing the 








clinical usefulness 








of PROM data 



EORTC QLQ-30 Three 



Doctors (n=10) No 



TLC-Chemo Alert 



Patients asked 
to report dally 
during the 
cycle and the 
alerts were 
sent to 
providers 



Doctors (n=2) Yes 



EORTC OOQ-C30 
Hospital Anxiety 
and Depression 
Scale (HADS) 



Regular clinic 
visit over an 
average of 6 
months 



Doctor (n=28) 



No 



Doctors given 
30-mlns 
training and 
patient 
mailed a 
leaflet 



Patients 
trained (10 
minutes TLC 
orientation) 



One to one 
training and 
manual 
provided 



Table 3 The characteristics of design and study quality (Continued) 



Basch et al. 
(2005) [41] 



Boyes et al. 
(2006) [35] 



Hoeskstra 
et al. (2006) 
[42] 



Komiblith 
et al. (2006) 
[43] 



USA (Memorial 
Sloan-Kettering 
Cancer Center, 
New York) 



Prospective pilot 
study of patient 
online self- 
reporting of 
toxicity 
symptoms 



80 patients 
diagnosed with 
a gynaecologic 
malignancy 
starting a new 
chemotherapy 
regimen 



Pilot controlled 
trial: Intervention 
/control 



80 cancer 
outpatients 
attending one 
cancer centre 



Australia 
(Centre for 
Health Research 
& Psycho- 
oncology, 
University of 
Newcastle) 



Netherlands RCT: Intervention 146 palliative 



(Academic 
Medical 
Centre, 
University of 
Amsterdam) 



USA (Dana- 
Farber Cancer 
Institute, 
Boston) 



group with 
symptom 
monitoring / 
control 



cancer patients 
recruited 
through two 
hospitals and 
local CPs 



RCT : Telephone 1 92 cancer 



Monitoring (TM) 
versus TM 
-tEducation 
Material (EM) 



patients with 
advanced 
disease and 
receiving active 
treatment 



Pattern of use of 
a Self-reported 
online Symptom 
Track and 
Reporting (STAR) 
system Patient 
impression of 
such system 
based on an exit 
questionnaire 
survey Clinician 
feedback 
(through survey 
and team 
debriefing) 

Patient symptoms 
Patient anxiety/ 
depression(HADS) 
Patient needs 
(Supportive Care 
Needs Survey 
[SCNS] 

Acceptability of 
intervention to 
patient and 
doctors 

10 symptoms 
from the 
Symptom 
Monitor Severity 
of the reported 
symptom 
(0-10 score) 

EORTC-QLQ-30 
HADS 



Symptom Track 
and Reporting 
(STAR) based on 
NCI CATAE system 



Any clinic visits 
during 8-wk 
study period 
(mean=3, 
range 1 -5) , 
also possible 
log in at home 
during the 
period 



Doctors and 
study team 
(n=unreported) 



Yes 



Training 
provide to 
patients but 
unreported 
to staff 



Symptoms, 
HADS SCNS 



1^' consultation Doctors (n=4) 



) patients: 



2 no 

719 



List of patients None 
needs 

accompanied 
by 

suggestions 
for 

appropriate 
referral 



Symptom Monitor Weekly self- CPs (98 times) No 



No 



Extensive 
Questionnaire 



EORTC-QLQ-30* 

HADS 

MOS-SS 

CDS (short form) 

QARSQ-PH 

UMPSI 

GSRE 

Patient Satisfaction 
with the Research 
Program BOMC 
test; 



assessed 

Symptom 

Monitor at 

home; 

Extensive 

questionnaire 

every 2-month 

Once a month 
over 6 months 



and medica 
specialists 
(96 times) 



Ontological 
nurses 



Yes 



Yes 



O 



Table 3 The characteristics of design and study quality (Continued) 



Basch et al. 
(2007) [44] 



USA (Memorial Prospective pilot 107 patients 



Rosenbloom 
et al. (2007) 
[34] 



Weaver et al. 
(2007) [45] 



Butt et al. 
(2008) [46] 



Sloan-Kettering study of a 



Cancer Center, 
New York) 



patient online 
self-reporting of 
toxicity 
symptoms 



Outcomes, 

Research and 

Education, 

Evanston 

Northwestern 

Healthcare 



UK (Oxford 
Radcliffe 
Hospitals 
NHS Trust) 



USA (Center 
on Outcomes, 
Research and 



interview and 
discussion / 
assessment 
control / 
standard care 



diagnosed with 

thoracic 

gynaecologic 

malignancy 

starting a new 

chemotherapy 

regimen 



USA (Center on RCT: Structured 21 3 patients 



A pilot study of 
novel mobile 
phone 
technology 



Prospective 
study to explore 
the longitudinal 



with advanced 
breast, lung or 
colorectal 
cancer 



6 colon cancer 
patients 



99 cancer 
patients with 
solid tumor of 



Feasibility/Pattern 
of use of a Self- 
reported online 
Symptom Track 
and Reporting 
(STAR) system 
Patient 
satisfaction 
survey 
(an exit 
questionnaire 
survey) Nursing 
survey (through 
an exit survey) 

Patient HRQoL 
(Functioning 
Living Index - 
Cancer [FLIC]) 
Patient affect 
(Brief Profile of 
Mood States 
[Brief POMS]) 
Patient 
satisfaction 
[PSQ-III] Clinical 
treatment 
changes as 
reported by nurse 
(supportive care 
changes, referrals, 
'other' clinical 
changes and 
changes in 
standard dose of 
chemotherapy as 
a result of PROs) 

Questionnaire on 
symptoms 
derived from the 
Common 
Terminology 
Criteria for 
Adverse Events 
(CTCAE) grading 
system 

FACT-G FACT- 
Fatigue subscale 
Brief Pain 



Symptom Track 
and Reporting 
(STAR) based on 
NCI CATAE 
system 



FACT-G and a 
single item 
asking patients 
whether a 
particular 
symptom or 
problem was 
better than, 
worse than, or 
as expected 



Questionnaire 
derived from the 
Common 
Terminology 
Criteria for 
Adverse Events 
(CTCAE) grading 
system 

FACT-G FACT- 
Fatigue subscale 
Brief Pain 



Any clinic visits 
during 42-wk 
study period 
(mean=12, 
range 1-40) , 
also possible log 
in at home 
during the 
period 



Nurses and 
study team 
(n=unreported) 



No 



Clinic visits at 
baseline ,and 1 
2,3 and 6 
months 



Twice daily 
during the 
chemotherapy 
circle (one 
morning, one 
evening) 



Baseline, 1 
month and 2 



Training 
provide to 
patients but 
unreported to 
staff 



Treating 
nurses (n=not 
reported) 



No 



No 



vvv 



Nurses 
(n= not 
reported) 



Yes 



Yes 



Doctors and 
nurses 



Table 3 The characteristics of design and study quality (Continued) 



Given et al. 
(2008) [47] 



Hilarius et al. 
(2008) [48] 



Mark et al. 
(2008) [49] 
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management 






USA( Michigan 


RCT: Nurse- 


129 breast 


Outcomes 


15 symptoms 


6 contacts or 


State University) 


Administrated 


cancer patients 


measured at 


(0-10 scale) 


self-reporting 




Symptom 




10-16 wks: 15 


Severity of the 


(1-4 wk, 6wk 




Managemerit 




symptoms 


symptoms 


8wk) 




\\ nAAJIvI/ Vi 




fD-in ^rp\\fA 

\\J I V jl^a 1 C J 








Al i1"r~ima1"Prl 












Tplpnhnnp 




M n n - rp*^ nn n '^pc; 

1 NWI I 1 CTjLyUI 1 JtTj 








jyl 1 ipLUI 1 1 




Ul iyi 1 ipLUI 1 lb 








^ /1 3 n a n fi m n"l" 
iviai laycri i ici IL 




Time to 








(ATSMl 

\r\ 1 Jivl/ 




1 CjLJUI I jC 








intervention 










Netherland 


A sequential 


10 nurses and 


Dartmouth 


EORTC QLQ-C30 


Four 


(Hospital 


cohort design 


219 patients 


Primary Care 


EORTC QLQ-BR23 


consecutive 


Pharmacy, Red 


with repeated 


cancer patients 


Cooperative 


EORTC QLQ-CR38 


visits after 


Cross Hospital, 


measures to 


with either 


Information 


E0RTCQLQ-LC13 


baseline for 


Beverwijk) 


evaluate the use 


adjuvant or 


Functional Health 




both pre 




of HRQL 


palliative 


Assessment 




(control arm) 




assessments in 


chemotherapy 


(COOPcharts) 




and post 




daily clinical 


in a community 


Patient 




(intervention 




oncology 


hospital 


Management 




arm) with a 




nursing practice 




extracted from 




two-month 








medical record 




'wash-out 








Patient 




period 



USA (Thomson 
Healthcare, 
Washington 
DC) 



A cross-sectional 
survey of the of 
both patients' 
and health 
professionals' 
experience; A 
before-after 



100 cancer 
patients and 92 
health 

professionals 
on the 

experience of 
The Patient 
Assessment, 



satisfaction 
(an exit survey 
based on PSQ, 
Form II) Patients' 
self-reported 
HRQL (SF-35, 
FACT-BCS, 
FACT-C, FACT-L) 
Nurse and patient 
evaluation of the 
intervention (an 
exit survey) 

Questionnaire 
survey of 1 02 
providers The 
patients 

satisfaction survey 
(n=100) including 
8-item on PCM 
200 patient chart 



PCM An 
education 
component (not 
reported in 
this study) 



Each visit to 
clinic 



Nurses or 
ATSM system 



Yes 



Yes 



Patients and 
nurses before 
consultations 



No 



Yes 



Clinicians No 
(n=unreported) 



n 



Table 3 The characteristics of design and study quality (Continued) 



Kearney et al. 
(2009) [50] 



UK (Cancer 
Care Research 
Centre, 
University of 
Stirling, Stirling) 



patient chart 
review 



RCT: Control 
group versus 
intervention 
group (mobile 
phone-based 
remote 
monitoring of 
symptoms) over 
five time points 



Carlson et al. 


Canada (Tom 


RCT: minimum 


(2010) [51] 


Baker Cancer 


screening 




Centre, 


(distress) / full 




University of 


screening / 




Calgary, 


Triage : full 




Alberta) 


screening + 






referring to 






appropriate 






services 


Dinkel et al. 


German 


Paired 


(2010) [52] 


(Department of 


comparison : a 




Psychotherapy 


computerised 




and 


and a paper 




Psychosomatic 


version of Stress 




Medicine, 


Index Radio 




Technische 


Oncology (SIRO) 




University 


tool Prospective 




Munchen) 


survey 



Care and 
Education (PACE) 
System, including 
PCIVl instrument 
and an education 
component 

56 patients with 
ung, breast or 
colorectal cancer 
for each group 
(total 112 
patients) 



585 breast 
cancer patients 
+ 549 lung 
cancer patients 



1 77 cancer 
patients in 
study 1,273 
cancer patients 
in study 2 
(n=142for 
computerised 
version and 
n=131 for 
paper version 
of SIRO) 27 
Patients, urses/ 
radiographs 
and 15 physicians 
evaluated the 
screening 
procedure 



reviews (1 00 
charts before and 
100 charts after 
the PACE system) 



;r version of 
electronic. Mobile 
phone-based 
Advanced 
Symptom 
Management 
Systems (ASyMS'') 
based on 
Common Toxicity 
Criteria Adverse 
Events (CTCAE) 
grading system 
and the 
Chemotherapy 
Symptom 
Assessment Scale 

Patient distress at 
3-month follow- 
up; Depression 
and anxiety at 3- 
month follow-up 



Agreement 
between 
computer and 
paper version of 
SIRO Patient 
satisfaction; Time 
need for both 
modes; Perceived 
utility; Perceived 
impact on 
communication; 
Perceived impact 
on patient 
outcome 



Mobile phone- 
based (ASyMS'') 
including 
chemotherapy- 
related morbidity 
of six common 
symptoms 
(nausea, vomiting, 
fatigue, mucositis, 
hand-foot 
syndrome and 
diarrhoea) 



Minimum 
screening: Distress 
thermometer (DT) 
Full screening: DT 
+ Psychological 
scan for cancer 
part C (PSSCAN) 



SIRO 



Five times 
including 
baseline and 
each of four 
chemotherapy 
cycles over a 
period of 1 4 
days 



Doctors only 
(n=unreported) 



Yes 



Yes 



Baseline 



Screening team Yes 

member 

(n=unreported) 



Yes 



Any visit 



Doctors and 
nurses 

(n=unreported) 



No 



No 



VV 



Table 3 The characteristics of design and study quality (Continued) 



Halkett et al. 
(2010) [11] 



(2011) [53] 



Velikova 

etal.(2010) 

[54] 



Australian 
(WA Centre for 
Cancer and 
Palliative Care, 
Curtin 
University) 



for Shared 
Decision 
IVIaking and 
Nursing 
Research, 
Oslo University 
Hospital, Oslo) 



UK (Cancer 
Research UK 
Clinical Centre 
- Leeds) 



Pilot study of 
using computer 
touch-screen 
technology 
to asses 
psychological 
distress in 
patients 



Ruland et al. Norway (Centre RCT: a 



computer- 
assisted, 
interactive 
tailored 
assessment 
(ITPA) with 
feedback vs ITPA 
only in oncology 
practice 



60 patients 
with various 
gynaecological 
cancers 



145 patients 
treated for 
leukaemia or 
lymphoma 



RCT: 

Intervention/ 
control- 
attention/control 
in a ratio of 2:1:1 



285 cancer 
patients 
commencing 
treatment at 
the Medical 
Oncology Clinic 
at St James 
Hospital 



Patient 

satisfaction with 
both touch- 
screen and paper 
questionnaire; 
Perceived utility 
of both modes by 
patients and 
health 

professionals 

Number of 
patient symptoms 
and problems 
addressed 
Changes in 
symptom distress 
Changes in 
patients' need for 
symptom 
management 
support over 
time, SF-35, 
Center for 
Epidemiological 
Studies 

Depression Scale 
(CES-D), IVledical 
Outcome Study 
Social Support 
Scale (MOS - SS) 

Medical Care 
Questionnaire 
(MCQ): 15-item 
three subscales: 
Communication, 
Coordination, 
Patient 
preferences 
Satisfaction with 
care Patients' and 
physicians' 
evaluation of the 
intervention K- 
index (Continuity 
of care: 
K={number of 
visits - number of 
doctors)/(number 
of visits -1). 



EORTC QLO-C30 
HADS The 
Supportive Care 
Needs Scale The 
Distress 
Thermometer 
Follow 

questionnaire 
survey on 
perceived utility 
of both modes 

Choice ITPA(19 

symptoms 

(0-4 scale on 

bothersome) 

and a severity 

scale 

of 0-10) 



EORTC QOQ-C30 
Hospital Anxiety 
and Depression 
Scale (HADS) 



Once 



Nurses and 
doctors 



Yes 



Yes 



Every inpatient 
admission with 
up to four 
follow-up visits 



Doctors 
and nurses 
(n=unreported) 



No (as see 
appropriate) 



Yes 



Regular clinic 
visit over an 
average of 6 
months 



Doctor (n=28) No 



One to one 
training and 
manual 
provided 



Table 3 The characteristics of design and study quality (Continued) 



Bainbridge 
etal.(2011) [55] 



Berry et al. 
(2011) [56] 



Cleeland 
et al. (201 1 
[20] 



Takeuchi 
etal. (2011) 
[57] 



Canada 


Survey on 


128 nurses. 


Perceptions of 


ESAS 


Every clinic visit 


Doctors and 


{Juravinski 


the utility of 


physicians, and 


use and utility of 






nurses 


Cancer Centre, 




allied health 


the Edmonton 
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Ontario) 






adopted by 
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centres since 














2007 








USA (Dana- 


RCT: 


660 cancer 


1. Audio-recorded 


Patient reported 


Every clinic visit 


Doctors 


Faber Cancer 


Intervention / 


patients with 


content of all 


symptoms and 


during the 


(n=76 principle 


Institute, 


Control 


various cancer 


communication 


quality-of-life 


study period 


physicians and 


Boston) 




diagnoses and 


between 


(SQLIs) from the 




other) or 






stages at two 


clinicians, 


Electronic 




incorporated 






institutions of a 


patients and 


Self-Report 




into charts 






comprehensive 


accompanying 


Assessment- 




(n=unreported) 






cancer centre 


friends or family 


Cancer (ESRA-C) 












members at each 














T2 visit (4-6 wks 














after the 














treatment) 














2Clinic visit 














duration 














... 
3. rhysician exit 














questionnaire 














survey 








USA (MD 


RCT: e-mail alert 


79 lung cancer 


1 . Four targeted 


Automated 


Twice weekly. 


Nurses 


Anderson 


of symptom to 


patients 


symptom: pain. 


telephone calls 


up to 4 wks 


(n=unreported) 


Cancer Center, 


patients' clinical 


receiving 


distress, disturbed 


(IVR system): 


after discharge 




The University 


team versus no 


thoracotomy 


sleep, and 


MDASI (13 






of Texas) 


e-mail alert 




shortness of 


common cancer 












breath. 


related 












constipation (no 


symptoms) 












fatigue as no 














effective 














response) 2. MD 














Anderson 














jyi I ipiui 1 1 




























at follow-up clinic 














visit 3. An exit 














questionnaire 














survey 








UK (St James's 


Longitudinal 


286 cancer 


Audio-recorded 


EORTC QOQ-C30 


Four 


Doctor (n=28) 


Institute of 


study of data as 


patients 


content of 


Hospital Anxiety 


consecutive 




Oncology, 


part of Velikova 


commencing 


Patient-physician 


and Depression 


visits from 




Leeds) 


et al. (2004, 


treatment at 


communication: 


Scale (HADS) 


baseline 






2010) RCT 


the Medical 


Longitudinal 









Yes 



Yes 



No 



Yes 



vvv 



Yes 



Training to 

patients 

provided 



No 



One to one 
training and 
manual 
provided 



VVV 



Table 3 The characteristics of design and study quality (Continued) 



Oncology Clinic 
at St James 
Hospital 



impact of PRO 
intervention; 
dynamics of 
communication; 
association 
between severity 
of symptoms/ 
functions and 
clinic discussion 



*: Four stars indicate a randomised trial or experimental study; 3 stars indicate a controlled trial, pre-post trial with control {controlled before-after trial), time series, or observational cohort with multivariable 
adjustment; 2 stars indicate a pre-post trial without control, observational cohort study without multivariable adjustment, cross-sectional study without multivariable adjustment, analysis of time trends without control, 
or well-designed qualitative study; and 1 star indicates a case series, other qualitative study, or survey (descriptive) study. 
V: Three checks indicate great weight in the stratum's body of evidence, 2 checks indicate moderate weight, and 1 check indicates little weight. 

+: EORTC QLQ-C30: European Organisation for Research and Treatment of Cancer Quality of Life (EORTC QLQ) Core Questionnaire; SF-36: The Short Form{36) Health Survey; FACT-G: Functional Assessment of Cancer 
Therapy-general; EQRTC QLQ-BR23: EQRTC QLQ Breast Cancer Scale; EQRTC QLQ-CR38: EQRTC QLQ Colorectal Cancer Scale; EORTC QLQ-LC; EORTC QLQ Lung Cancer Scale; NCI CATAE: National Cancer Institute 
Common Terminology Criteria for Adverse Events; FACT-BCS: Functional Assessment for Cancer Therapy-Breast Cancer Subscale; FACT-C: Functional Assessment of Cancer Therapy-Colorectal Quality of Life Instrument; 
FACT-L: Functional Assessment of Cancer Therapy-Lung Cancer Subscale; MOS-SS: Medical Outcomes Study Social Support Survey; CDS: Geriatric Depression Scale (short form); QARSQ-PH: Physical Health subscale of 
the Older American Resources and Services Questionnaire (OARSQ); UMPSI: Utilisation of Mental Health and Psychosocial Services Instrument; GSRE: Geriatric Schedule of Recent Experience Instrument; 
?: No data or unable to classify data available. 



Chen ef al. BMC Health Services Research 201 3, 1 3:21 1 
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Table 4 The components, rating criteria and symbol, and categories used in summarising the study evidence in the 
current study 





Domain 1 


Domain 2 


Global (GRADE) 


Decision 
Components 


Subject of public reporting (or 
study population) and study 
participants (sample) 


Types of study (i.e. study designs) 


Components from Domain 1 & 2 as well as 
implementation and adherence to intervention, dose- 
response gradient, precision and validity of the 
outcomes, uncertainty of direction of the resu ts. 


Rating criteria 


How well does the study sample 
represent the study population? 


How strong is the study design 
both in terms of its external and 
internal validity? 


How much weight does the current study add to the 
evidence-base taking into considerations of all the 
components above? 


Symbol used & 
categories of 
rating 


1*: no overlap 


1*: weakest design 


V: little weight 




2*: modest overlap 


2*: moderate design 


VV: moderate weight 




3*: large overlap 


3*: strong design 


VVV: great weight 




4*: complete overlap 


4*: strongest design 





reviews, it was listed in the summation tables for the 
purpose of comparison. 

Overview of study quality 

There has been a marked increase in the volume and 
quality of the studies published recently in this area. Of 
the 16 randomised controlled trials included in this re- 
view, 7 were published between 2010 and 2011. The 
quality of studies published since 2010 is also demon- 
strably improved with much larger sample sizes, includ- 
ing 3 trials [54,57,61] with a sample size greater than 
200 and 2 trials with a sample size over 580 [51,56]. 

However, despite the increased volume and improved 
quality of the studies, there remains a lack of large 
cluster randomised controlled design studies, as 
recommended by Payers [62] who argues that cluster 
RCTs are well suited to overcome the limitations of sim- 
ple RCTs. It is well-known that system intervention tri- 
als such as routine collection of PROs, and feedback to 
the clinicians and systems, are prone to cross- 
contamination and to introducing investigator and par- 
ticipant biases. Two recently published studies [54,57] 
were the continuation of an earlier study published by 
Velikova et al. (2004) [36]. Most studies reviewed did 
not systematically examine outcomes and mechanisms, 
and placed more emphasis on processes rather than out- 
come measures [25]. All studies were conducted in a 
limited setting (often in a single centre) thus restricting 
the generalisation of the findings. 

No studies have adopted a comprehensive theoretical 
model and framework, despite the repeated demand 
from leading researchers in the area [25,63-65]. All stud- 
ies focused on the patient and health professional level 
within the clinic setting. No study to date has examined 
the impact of collecting PROs on health care organisa- 
tions, health system improvement, quality improvement 
or population health at a system or societal level. 



Overview of study findings 

Impact on patient-provider communication 

Across the 27 studies included in this review, 4 studies 
[39,47,51,53] did not examine or report the effect of a rou- 
tinely collected PRO on patient-provider communication. 
Among the 23 studies that did report such an impact, 21 
studies (91.3%) reported a positive effect which included 
well-designed and conducted large RCTs [33,36,37,54,56,57]. 
One study reported no significant improvement of patient- 
provider communication possibly due to a lower severity 
level of cancer patients (only 37% of patients received anti- 
cancer therapy, hence the reduced need for communication 
for the treatment) [38]. Another study reporting a negative 
effect had an already high communication level at baseline 
(hence a ceiling effect leaving little room for further 
improvement) [34]. 

Impact on monitoring treatment response 

Despite most of included 27 studies did not explicitly state 
their study objectives as to examine the impact on moni- 
toring treatment response, 11 of the 27 studies did report 
an outcome (Table 3) [16,20,36,41,45-51]. All 11 studies 
found a strong or modest effect of implementing PROs on 
the increased monitoring activities of treatment response. 
The strongest effect occurred in the studies that focused 
on the monitoring of patient symptoms, side effects and 
toxicity during and after chemotherapy for the outpatients. 
In particular, the real-time, patient reported symptoms 
and toxicity (through innovative mobile phone-based, 
web-based or IVR systems) significantly improved the 
monitoring of treatment response. 

Impact on detecting unrecognised problems 

Although the idea of routinely collected PROs may provide 
better opportunities for services providers (as well as pa- 
tients) to detect unrecognised problems through growing 
awareness, improved communication and monitoring seems 



Table 5 The impact and effect sizes of the studies on patients, care providers and organisations* 



Studies Doctor-patient IVIonitor Detect Changes Changes to Improved Improved Feasibility Moderating and 

communication treatment unrecognised to patient health behaviour patient patient health outcomes of the subgroup effect 

response problems management satisfaction implementation 



Trowbridge 
etal. (1997)[39] 

Tazenzer et al. +++ 
(2000)03] 

McLachlan -(no time 
etal. (2001 )[38] differences in 
consultation 
between two 
arms) 



Detmar et al. 
(2002) [3 7] 



Mooney et al. 
(2002) [16] 

Velikova et al. 
(2004)[36] 



Basch et al. 
(2005) [41] 



Boyes et al. 
(2006)[35] 



Hoekstra et al. 
(2006) [42] 



-H-+ (10 out of 
12 HRoL 
measures, 
especially on 
soda 

functioning and 
fatigue) 



+ (50% 
oncologists in 
intervention 
group talked 
with patients) 

4-/-(0nly 18% 
patients used it 
enhancing 
communication) 



-(only 37% 

patients 

receiving 

anticancer 

therapy at 

baseline) 



++ (64% 
encounters 
involving 
referring to 
HRoL by 
physicians) 



+ (but no change +++ 
in PIVll) 



+ (increased 
patient 

counselling) +( 
25% with family 
members and 
primary care 
physicians) 



+ (emotiona 
support) 

++ (information 
sharing & 
communication) 



+ (SF-36) 



-(possible due to 
simple coding 
between two arms) 
4-(contributed to 
patient 

management in 
11% of encounters 
intervention arm). 



-H-(overall quality 
of life and 
emotiona 
functioning) 



++ (response 
rate 70%) 



-H- (65% patient 
log in before 
any verbal 
encouragement) 



++ (fewer 
deliberating 
symptoms) - 
(anxiety and 
depression) 

++ (lower 
prevalence in 9 
out of 1 0 
symptoms; 
deteriorated less 
in 8 out of 10 
symptoms) 



+ (on high BDI score 
subgroup) 



+ (before-after 
improvement by 
intervention group) 



+ (more discussion of 
HRoL subgroup had 
better outcome within 
intervention group) 



The beneficial effects 
were pronounced in the 
deteriorated group. 



Table 5 The impact and effect sizes of the studies on patients, care providers and organisations* (Continued) 



Korniblith et al. +++ (both arms) 
(2006)[43] 



Basch et al. + 
(2007) [44] 



Rosenbloom -(Possible 
et al. {2007)[34] Ceiling effect) 

Weaver et al. + (nurse-patient + 

(2007) [45] communication) 

Butt et al. -H- + 

(2008) [46] 

Given et al. -I- 
(2008) [47] 



++ (more 
from TIVI-fEM 
arm) 



++ (both arms) 



Hilarius et al. ++ 
(2008) [48] 

Mark et al. -H- 

(2008) [49] 

Kearney et al. ++ 

(2009) [50] 

Carlson et al. 

(2010) [51] 



Dinkel et al. 

(2010)[52] 



Halkett et al. 
(2010)[11] 



+ (around 25% 
of doctors) 



+ (10% patients 
reported changed 
outcomes) 



++ (better in TM ++ 
+EM arm - 
reduction of 
psychological 
distress) 



++ (can be 
improved 
through 
reminder) 



++ (ATSM more 
likely to 
generated 
responses in 
symptom 
management and 
required less time 
to do so) 



++ (Fatigue) 

+++ (distress) ++ 
(decreased 
depression and 
anxiety related to 
referral to 
services) 



No effect even among 
the most highly 
distressed patients 



+ (Compared with 
patients receiving 
combination 
chemotherapy protocols, 
those patients treated 
with single agent had 
greater response and 
shorter time to 
response) 



+ (patients is 
generally happy 
with both 
methods) -(Health 
professionals 
found some 
issues) 



+/- (some issues 
identified but 
nothing 

fundamental and 
patients were 
generally happy) 



Table 5 The impact and effect sizes of the studies on patients, care providers and organisations* (Continued) 



Ruland et al. 
(2010)[53] 




++ 


++ 


++ 


++ 




Velikova et al. 
(2010)[54] 


++ 




(no difference in 
coordination of 
care & 'preferences 
to see usual 
doctor' subscale) 


++(86% in 
intervention vs 
29% in the 
attention- 
control group) 


++ 




Bainbridge 
etal. (2011) [55] 


+ + 




+ 




++ 


+ 89% of nurses and 
55%) of physicians 
referred to the ESAS in 
clinics 'always' or ' most 
of the time' 


Berry et al. 
(2011)[56] 


++ (25% 
physician 
explicitly 
referred to SQLI 
summary) 








++ 


++ (the treatment effect 
on communication is 
evident on over 
threshold group on 
cognitive function, 
impact on sex and social 
function) 


Cleeland et al. 
(2011)[20] 


++ + 


+ 


+ 


+ ++ 


++ 




Takeuchi et al. 
(2011)[57] 


++ (on 
symptom but 
not function) 








++ 





Note: +++ very strong effect; ++ strong effect; + some effect; +/- uncertain effect; - No effect; blank : untested or reported; 

*: Impacts on quality improvement, increased transparency, accountability, public reporting, better population and system performance (monitoring, planning, financing, evaluating, etc) were not listed due to lack of 
data. 
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Records identified tiirougli database 
searching 
(n= 1062) 



Additional records identified 
through other sources 
(n=182) 



Records after duplicates removed 
(n =62) 



Records screened 
(n=1182) 



Full-text articles assessed 
for eligibility 
(n=62) 



Studies included in 
qualitative synthesis 
(n = 27) 



Records excluded 
(n=1120) 



Full-text articles excluded, 
with reasons 
(n = 35) 

• description of 
information technique 
of PROs collection 
only(16) 

• not including 
sufficient study 
endpoint(s) (12) 

• not having sufficient 
quantitative analyses 
to enable assessment 
(7) 



Figure 3 PRISMA Flow Diagram illustrating the systematic review process from electronic searching through to study inclusion. 



intuitively plausible, only 16 out of 27 studies reported some 
results related to the detection of unrecognised problems 
(Table 3). Amongst the 16 studies, 15 studies [16,20,33,36, 
37,39,41,43,45-50,53] reported either a strong or moderate 
positive impact on detecting unrecognised problems. How- 
ever, a study by McLachlan and colleagues [38] did not find 
any difference between the intervention arm and control 
arm. 

Impact on changes to patient health behaviour 

No study to date has provided a systematic evaluation 
on the impact of collecting PROs on changes to patient 
health behaviour. It is unknown whether and how pa- 
tient health behaviours have been changed. 

Impact on changes to patient management 

Amongst 17 studies that provided some results of changes 
to patient management, 13 studies [11,20,33,37,39, 
45,46,48-50,52,53,55] reported either a strong or modest 
positive effect on the changes to patient management 
whilst 4 studies [34-36,54] found no such effect. However, 
it is worth noting that 10 studies did not provide any infor- 
mation about the changes to patient management and 



there were often less complete descriptions of the results 
on patient management when reported. 

Impact on patient satisfaction 

Among the 16 studies that reported results related to 
the impact on patient satisfaction, 13 studies [11,16,20, 
37,41,43-46,48,49,52,54] reported a very strong to mod- 
erate positive effect on improved patient satisfaction. For 
the three studies [33,34,38] that did not find such a posi- 
tive effect, one study [33] reported a possible ceiling ef- 
fect meaning that both the intervention group and 
control group had a very high baseline patient satisfac- 
tion level potentially impeding any demonstration of a 
significant difference between two arms during the 
follow-up period. 

Impact on health outcomes 

Amongst the 15 studies that reported some results re- 
lated to the impact on health outcomes, 13 studies 
[20,35-37,39,41-43,45,47,50,51,53] reported some posi- 
tive improvement, ranging from modest to strong, while 
two studies [34,38] failed to find any such effect. It ap- 
pears that symptoms, side effects and toxicity are most 
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likely to be improved, followed by emotional wellbeing. 
There is little evidence on the improvement of both 
overall HRQOLs as well as social wellbeing. 

Impact on quality improvement, transparency, accountability 
and public reporting, and on better system performances 
(monitoring, planning, financing, evaluating, responding) 

No study to date has provided a meaningful, explicit 
framework nor relevant evidence on these endpoints. 

Overall strength and direction of evidence 

Overall, there is strong evidence supporting the notion 
that routinely collected PROs, with feedback, improves 
patient-provider communication and increases patient 
satisfaction (Table 6). There is some evidence to support 
the notion that it improves the monitoring of treatment 
responses and detection of unrecognised problems, and 
there is weak but positive evidence that, over time, it 
leads to changes in patient management. Despite some 
encouraging results, there is still a great degree of uncer- 
tainty regarding the impact of routinely collected PROs, 
with feedback, on patient health outcomes. There is little 
or no evidence that it has led to significant positive im- 
provements in quality improvement, transparency, ac- 
countability, and public reporting, or in system 
performance at a population health or societal level. 
Apart from clinical trials and clinical practice, its impact 
on health services research and population health is 
largely unknown. 

Potential moderating factors and links between routine 
PRO collection (with feedback) and patient outcomes 

Although the evidence is limited, it appears that routine 
collected PROs with sufficient intensity of feedback 
(multiple times over a sustained period of time) 



Table 6 The overall strength and direction of evidence 



Results 


Strength and 
direction of evidence 


Patient-provider communication 


+++ 


Monitor treatment response 


++ 


Detect unrecognised problems 


++ 


Changes to patient health behaviour 


n/a 


Changes to patient management 


+ 


mproved patient satisfaction 


+++ 


Improved health outcomes 


+/0 


Strong & effective quality improvement 


n/a 


ncreased transparency, accountability, public 


n/a 


reporting 




Better system performance (monitoring. 


n/a 


planning, financing, evaluating, responding) 





Note: -t-F-f-F: the strongest positive effect; xxxx: the strongest negative effect; 
n/a: not available; 0: mutual (no significant effect). 



[13,39,44,54], targeting multiple stakeholders (doctors, 
nurses, allied health workers, as well as patients) [35,52] 
with simple, clear, graphical and longitudinal meaningful 
interpretation of the results, and providing sufficient 
training for both health professionals and patients 
[20,57], are critical links between an intervention and 
the intended outcomes. There is also evidence to suggest 
that for some complex issues such as depression and 
low social functioning, routine screening and feedback 
may need to be integrated with other strategies such as 
decision-making aids, education, clear management 
plans and clinical pathways including referrals, in order 
to change patient outcomes [43,49,51]. There is prelim- 
inary evidence that some of the impacts of PROs may be 
more pronounced amongst subgroups with more severe 
problems at baseline (e.g. depression, symptoms) 
[38,42,65]. More studies are needed to fully understand- 
ing these mediating and moderating effects. 

Discussion 

There is very strong evidence in supporting the notion 
that routine collected PROs with timely feedback en- 
hances patient-provider communication. This current 
study finding of a positive effect on patient-provider 
communication is consistent with previous reviews 
conducted in both cancer [26] and non-cancer settings 
[25,27,28]. There is also strong evidence to support the 
notation that routine collected PROs significantly im- 
proved the monitoring of treatment response. 

There is reasonably strong evidence in supporting the 
notation that routine collected PROs are helpful in identi- 
fying unrecognised problems in a large variety of settings. 
Within studies that reported, to some extent, the results 
related to unrecognised problems, there is a need for the 
development of more comprehensive and valid measures. 
Such a change in the measures would contribute towards 
understanding specifically the PROs' impact on identifying 
the underreported and unrecognised problems for differ- 
ent cancer patients at different settings. 

Overall, there is reasonable evidence in favouring the 
hypothesis that implementing a routine collected PROs 
system brings positive changes to patient management in 
the settings where a patient management plan is inte- 
grated with a routine collection of PROs. It appears that 
the simple routine feedback of PROs may not be sufficient 
to improve patient management and outcomes [48] . Other 
necessary resources may be needed such as education, re- 
ferral services and a detailed patient management plan fol- 
lowing the PROs [43]. There is also a need to develop 
better measures of change to patient management as it is 
often complex and difficult to quantify [57]. 

There is strong evidence to support the notation that 
routine collected PROs with timely feedback significantly 
enhance patient's experience and satisfaction. There may 
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be other improved experience and satisfaction in other 
stakeholders such as patients' family members, caregivers, 
as well as health professionals that were not measured or 
unreported. Future research into furthering the under- 
standing of stakeholder experience after implementing 
routine collected PROs would be desirable. 

Although positive evidence in supporting the notion 
that routine collected PROs may improve health out- 
comes is weak, this finding needs to be confirmed by 
better designed studies covering a large set of well- 
developed outcome measures. There is also a need to 
understand the impact on long-term health outcomes 
such as survival rate. Most of the studies included in this 
review did not focus on health outcomes and some of 
the positive improvement on the outcomes only oc- 
curred on selective measures. It is not clear how these 
positive improvements can be generalised across differ- 
ent settings. 

There is a variety of models on how to routinely col- 
lect PROs and how to feed back the data to different 
stakeholders. Given that cancer patients are vastly differ- 
ent given their background, type and stage of cancer, 
prognosis, treatment, and the positions at the life course 
continuum, precaution should be exercised when 
attempting to apply the general observation above to 
each and every different setting. For example, recent 
studies demonstrated a positive impact of routine col- 
lected PROs on symptom control through either web- 
based or mobile phone based approach. However, such 
positive impacts were less pronounced on HRQOL. 

Limitations of the current review 

Our review has several limitations. First, there was no 
attempt made to contact the authors to ask for potential 
unpublished data on the topic. Thus, there may be 
chance of missing some grey literature or the studies 
that under preparation for publication. Second, given the 
multitude of endpoints included, and different types of 
studies involved, the assessment of eligibility for inclu- 
sion of potential studies required some degree of sub- 
jective judgement. Third, our application of GRADE 
system was rather simplistic restricted by large number 
of endpoints and variability of studies included. These 
limitations may give rise to some uncertainty in terms of 
synthesis of the results. Fourth, our study follows a sys- 
tematic review approach with inclusion of both experi- 
mental trials and quantitative observational studies. 
However, we did not include qualitative studies in our 
review which may provide additional insight into the 
questions raised. This is particularly relevant with re- 
spect to questions 2 and 3 as there was very little quanti- 
tative evidence from the included studies. It is important 
to note that despite efforts to formulate the review end- 
points based on solid and well-established causal and 



theoretical frameworks (providing insight into not only 
if but also how the introduction of PROs affects patient 
outcomes), the causal mechanisms and process end- 
points included in the current review are by no means 
exhaustive. There may be other important causal mecha- 
nisms that could be benefited from a realist review ap- 
proach [66]. 

Conclusions 

There is growing evidence supporting the routine collec- 
tion of PRO to enable better and patient-centred care, 
especially in cancer settings. Despite the strong evidence 
in supporting the notion that the well-implemented rou- 
tine collection of PROs enhances patient-provider com- 
munication and improves patient satisfaction, and 
growing evidence supporting ideas that it also improves 
the monitoring of treatment response and the detection 
of the unrecognised problems, the evidence-base was 
weak for its impact on changes to patient management 
and improved health outcomes and non-existent for 
changes to patient health behaviour, strong and effective 
quality improvement, increased transparency, account- 
ability, public reporting and better health care system 
performance. These evidence gaps require further com- 
mitted and well-planned research in addition to the 
well-accepted PROs. Decision-making agencies have 
been well positioned for leverage on the rapid advance- 
ment of different PRO models, the application of the 
item response theory and computer adapted test in de- 
veloping PROs, and on the acceptance of such technol- 
ogy by patients and health professionals over the last 
decade. The real-time and routinely collected PROs will 
enable the development of a rapid learning health sys- 
tem with the potential to advance our knowledge of drug 
development, unsurpassed models of cancer patient care 
and a more patient-centred health care system. 
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